<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Basic concepts of the homography explained with code</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../d9/d97/tutorial_table_of_content_features2d.html">2D Features framework (feature2d module)</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Basic concepts of the homography explained with code </div>  </div>
</div><!--header-->
<div class="contents">
<div class="toc"><h3>Table of Contents</h3>
<ul><li class="level1"><a href="#tutorial_homography_Introduction">Introduction </a><ul><li class="level2"><a href="#tutorial_homography_Basic_theory">Basic theory </a><ul><li class="level3"><a href="#tutorial_homography_What_is_the_homography_matrix">What is the homography matrix?</a></li>
<li class="level3"><a href="#tutorial_homography_How_the_homography_transformation_can_be_useful">How the homography transformation can be useful?</a></li>
</ul>
</li>
<li class="level2"><a href="#tutorial_homography_Demonstration_codes">Demonstration codes </a><ul><li class="level3"><a href="#tutorial_homography_Demo1">Demo 1: Pose estimation from coplanar points</a></li>
<li class="level3"><a href="#tutorial_homography_Demo2">Demo 2: Perspective correction</a></li>
<li class="level3"><a href="#tutorial_homography_Demo3">Demo 3: Homography from the camera displacement</a></li>
<li class="level3"><a href="#tutorial_homography_Demo4">Demo 4: Decompose the homography matrix</a></li>
<li class="level3"><a href="#tutorial_homography_Demo5">Demo 5: Basic panorama stitching from a rotating camera</a></li>
</ul>
</li>
<li class="level2"><a href="#tutorial_homography_Additional_references">Additional references </a></li>
</ul>
</li>
</ul>
</div>
<div class="textblock"><p><b>Prev Tutorial:</b> <a class="el" href="../../dc/d16/tutorial_akaze_tracking.html">AKAZE and ORB planar tracking</a></p>
<table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 3.0 </td></tr>
</table>
<h1><a class="anchor" id="tutorial_homography_Introduction"></a>
Introduction </h1>
<p>This tutorial will demonstrate the basic concepts of the homography with some codes. For detailed explanations about the theory, please refer to a computer vision course or a computer vision book, e.g.:</p><ul>
<li>Multiple View Geometry in Computer Vision, <a class="el" href="../../d0/de3/citelist.html#CITEREF_HartleyZ00">[103]</a>.</li>
<li>An Invitation to 3-D Vision: From Images to Geometric Models, <a class="el" href="../../d0/de3/citelist.html#CITEREF_Ma:2003:IVI">[158]</a></li>
<li>Computer Vision: Algorithms and Applications, <a class="el" href="../../d0/de3/citelist.html#CITEREF_RS10">[236]</a></li>
</ul>
<p>The tutorial code can be found here <a href="https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/Homography">C++</a>, <a href="https://github.com/opencv/opencv/tree/master/samples/python/tutorial_code/features2D/Homography">Python</a>, <a href="https://github.com/opencv/opencv/tree/master/samples/java/tutorial_code/features2D/Homography">Java</a>. The images used in this tutorial can be found <a href="https://github.com/opencv/opencv/tree/master/samples/data">here</a> (<code>left*.jpg</code>).</p>
<h2><a class="anchor" id="tutorial_homography_Basic_theory"></a>
Basic theory </h2>
<h3><a class="anchor" id="tutorial_homography_What_is_the_homography_matrix"></a>
What is the homography matrix?</h3>
<p>Briefly, the planar homography relates the transformation between two planes (up to a scale factor):</p>
<p class="formulaDsp">
\[ s \begin{bmatrix} x^{&#39;} \\ y^{&#39;} \\ 1 \end{bmatrix} = H \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} h_{11} &amp; h_{12} &amp; h_{13} \\ h_{21} &amp; h_{22} &amp; h_{23} \\ h_{31} &amp; h_{32} &amp; h_{33} \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \]
</p>
<p>The homography matrix is a <code>3x3</code> matrix but with 8 DoF (degrees of freedom) as it is estimated up to a scale. It is generally normalized (see also <a class="el" href="../../d9/dab/tutorial_homography.html#lecture_16">1</a>) with \( h_{33} = 1 \) or \( h_{11}^2 + h_{12}^2 + h_{13}^2 + h_{21}^2 + h_{22}^2 + h_{23}^2 + h_{31}^2 + h_{32}^2 + h_{33}^2 = 1 \).</p>
<p>The following examples show different kinds of transformation but all relate a transformation between two planes.</p>
<ul>
<li>a planar surface and the image plane (image taken from <a class="el" href="../../d9/dab/tutorial_homography.html#projective_transformations">2</a>)</li>
</ul>
<div class="image">
<img src="../../homography_transformation_example1.jpg" alt="homography_transformation_example1.jpg"/>
</div>
<ul>
<li>a planar surface viewed by two camera positions (images taken from <a class="el" href="../../d9/dab/tutorial_homography.html#szeliski">3</a> and <a class="el" href="../../d9/dab/tutorial_homography.html#projective_transformations">2</a>)</li>
</ul>
<div class="image">
<img src="../../homography_transformation_example2.jpg" alt="homography_transformation_example2.jpg"/>
</div>
<ul>
<li>a rotating camera around its axis of projection, equivalent to consider that the points are on a plane at infinity (image taken from <a class="el" href="../../d9/dab/tutorial_homography.html#projective_transformations">2</a>)</li>
</ul>
<div class="image">
<img src="../../homography_transformation_example3.jpg" alt="homography_transformation_example3.jpg"/>
</div>
<h3><a class="anchor" id="tutorial_homography_How_the_homography_transformation_can_be_useful"></a>
How the homography transformation can be useful?</h3>
<ul>
<li>Camera pose estimation from coplanar points for augmented reality with marker for instance (see the previous first example)</li>
</ul>
<div class="image">
<img src="../../homography_pose_estimation.jpg" alt="homography_pose_estimation.jpg"/>
</div>
<ul>
<li>Perspective removal / correction (see the previous second example)</li>
</ul>
<div class="image">
<img src="../../homography_perspective_correction.jpg" alt="homography_perspective_correction.jpg"/>
</div>
<ul>
<li>Panorama stitching (see the previous second and third example)</li>
</ul>
<div class="image">
<img src="../../homography_panorama_stitching.jpg" alt="homography_panorama_stitching.jpg"/>
</div>
<h2><a class="anchor" id="tutorial_homography_Demonstration_codes"></a>
Demonstration codes </h2>
<h3><a class="anchor" id="tutorial_homography_Demo1"></a>
Demo 1: Pose estimation from coplanar points</h3>
<dl class="section note"><dt>Note</dt><dd>Please note that the code to estimate the camera pose from the homography is an example and you should use instead <a class="el" href="../../d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d">cv::solvePnP</a> if you want to estimate the camera pose for a planar or an arbitrary object.</dd></dl>
<p>The homography can be estimated using for instance the Direct Linear Transform (DLT) algorithm (see <a class="el" href="../../d9/dab/tutorial_homography.html#lecture_16">1</a> for more information). As the object is planar, the transformation between points expressed in the object frame and projected points into the image plane expressed in the normalized camera frame is a homography. Only because the object is planar, the camera pose can be retrieved from the homography, assuming the camera intrinsic parameters are known (see <a class="el" href="../../d9/dab/tutorial_homography.html#projective_transformations">2</a> or <a class="el" href="../../d9/dab/tutorial_homography.html#answer_dsp">4</a>). This can be tested easily using a chessboard object and <code><a class="el" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a" title="Finds the positions of internal corners of the chessboard. ">findChessboardCorners()</a></code> to get the corner locations in the image.</p>
<p>The first thing consists to detect the chessboard corners, the chessboard size (<code>patternSize</code>), here <code>9x6</code>, is required:</p>
<div class="fragment"><div class="line">    vector&lt;Point2f&gt; corners;</div><div class="line">    <span class="keywordtype">bool</span> found = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">findChessboardCorners</a>(img, patternSize, corners);</div></div><!-- fragment --> <div class="image">
<img src="../../homography_pose_chessboard_corners.jpg" alt="homography_pose_chessboard_corners.jpg"/>
</div>
<p>The object points expressed in the object frame can be computed easily knowing the size of a chessboard square:</p>
<div class="fragment"><div class="line">        <span class="keywordflow">for</span>( <span class="keywordtype">int</span> i = 0; i &lt; boardSize.height; i++ )</div><div class="line">            <span class="keywordflow">for</span>( <span class="keywordtype">int</span> j = 0; j &lt; boardSize.width; j++ )</div><div class="line">                corners.push_back(<a class="code" href="../../dc/d84/group__core__basic.html#ga3d79ceeb4419bccd0308dfdf1cd31435">Point3f</a>(<span class="keywordtype">float</span>(j*squareSize),</div><div class="line">                                          <span class="keywordtype">float</span>(i*squareSize), 0));</div></div><!-- fragment --><p> The coordinate <code>Z=0</code> must be removed for the homography estimation part:</p>
<div class="fragment"><div class="line">    vector&lt;Point3f&gt; objectPoints;</div><div class="line">    calcChessboardCorners(patternSize, squareSize, objectPoints);</div><div class="line">    vector&lt;Point2f&gt; objectPointsPlanar;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 0; i &lt; objectPoints.size(); i++)</div><div class="line">    {</div><div class="line">        objectPointsPlanar.push_back(<a class="code" href="../../dc/d84/group__core__basic.html#ga7d080aa40de011e4410bca63385ffe2a">Point2f</a>(objectPoints[i].x, objectPoints[i].y));</div><div class="line">    }</div></div><!-- fragment --><p> The image points expressed in the normalized camera can be computed from the corner points and by applying a reverse perspective transformation using the camera intrinsics and the distortion coefficients:</p>
<div class="fragment"><div class="line">    FileStorage fs( <a class="code" href="../../d6/dba/group__core__utils__samples.html#ga3a33b00033b46c698ff6340d95569c13">samples::findFile</a>( intrinsicsPath ), FileStorage::READ);</div><div class="line">    Mat cameraMatrix, distCoeffs;</div><div class="line">    fs[<span class="stringliteral">&quot;camera_matrix&quot;</span>] &gt;&gt; cameraMatrix;</div><div class="line">    fs[<span class="stringliteral">&quot;distortion_coefficients&quot;</span>] &gt;&gt; distCoeffs;</div></div><!-- fragment --> <div class="fragment"><div class="line">    vector&lt;Point2f&gt; imagePoints;</div><div class="line">    <a class="code" href="../../d9/d0c/group__calib3d.html#ga55c716492470bfe86b0ee9bf3a1f0f7e">undistortPoints</a>(corners, imagePoints, cameraMatrix, distCoeffs);</div></div><!-- fragment --><p> The homography can then be estimated with:</p>
<div class="fragment"><div class="line">    Mat H = <a class="code" href="../../d9/d0c/group__calib3d.html#ga4abc2ece9fab9398f2e560d53c8c9780">findHomography</a>(objectPointsPlanar, imagePoints);</div><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;H:\n&quot;</span> &lt;&lt; H &lt;&lt; endl;</div></div><!-- fragment --><p> A quick solution to retrieve the pose from the homography matrix is (see <a class="el" href="../../d9/dab/tutorial_homography.html#pose_ar">5</a>):</p>
<div class="fragment"><div class="line">    <span class="comment">// Normalization to ensure that ||c1|| = 1</span></div><div class="line">    <span class="keywordtype">double</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga4e556cb8ad35a643a1ea66e035711bb9">norm</a> = <a class="code" href="../../d0/de1/group__core.html#ga9070b6a3f093dd952d973819b06f4906">sqrt</a>(H.at&lt;<span class="keywordtype">double</span>&gt;(0,0)*H.at&lt;<span class="keywordtype">double</span>&gt;(0,0) +</div><div class="line">                       H.at&lt;<span class="keywordtype">double</span>&gt;(1,0)*H.at&lt;<span class="keywordtype">double</span>&gt;(1,0) +</div><div class="line">                       H.at&lt;<span class="keywordtype">double</span>&gt;(2,0)*H.at&lt;<span class="keywordtype">double</span>&gt;(2,0));</div><div class="line"></div><div class="line">    H /= <a class="code" href="../../dc/d84/group__core__basic.html#ga4e556cb8ad35a643a1ea66e035711bb9">norm</a>;</div><div class="line">    Mat c1  = H.col(0);</div><div class="line">    Mat c2  = H.col(1);</div><div class="line">    Mat c3 = c1.cross(c2);</div><div class="line"></div><div class="line">    Mat tvec = H.col(2);</div><div class="line">    Mat R(3, 3, <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga30a562691cc5987bc88eb7bb7a8faf2b">CV_64F</a>);</div><div class="line"></div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0; i &lt; 3; i++)</div><div class="line">    {</div><div class="line">        R.at&lt;<span class="keywordtype">double</span>&gt;(i,0) = c1.at&lt;<span class="keywordtype">double</span>&gt;(i,0);</div><div class="line">        R.at&lt;<span class="keywordtype">double</span>&gt;(i,1) = c2.at&lt;<span class="keywordtype">double</span>&gt;(i,0);</div><div class="line">        R.at&lt;<span class="keywordtype">double</span>&gt;(i,2) = c3.at&lt;<span class="keywordtype">double</span>&gt;(i,0);</div><div class="line">    }</div></div><!-- fragment --> <p class="formulaDsp">
\[ \begin{align*} \boldsymbol{X} &amp;= \left( X, Y, 0, 1 \right ) \\ \boldsymbol{x} &amp;= \boldsymbol{P}\boldsymbol{X} \\ &amp;= \boldsymbol{K} \left[ \boldsymbol{r_1} \hspace{0.5em} \boldsymbol{r_2} \hspace{0.5em} \boldsymbol{r_3} \hspace{0.5em} \boldsymbol{t} \right ] \begin{pmatrix} X \\ Y \\ 0 \\ 1 \end{pmatrix} \\ &amp;= \boldsymbol{K} \left[ \boldsymbol{r_1} \hspace{0.5em} \boldsymbol{r_2} \hspace{0.5em} \boldsymbol{t} \right ] \begin{pmatrix} X \\ Y \\ 1 \end{pmatrix} \\ &amp;= \boldsymbol{H} \begin{pmatrix} X \\ Y \\ 1 \end{pmatrix} \end{align*} \]
</p>
<p class="formulaDsp">
\[ \begin{align*} \boldsymbol{H} &amp;= \lambda \boldsymbol{K} \left[ \boldsymbol{r_1} \hspace{0.5em} \boldsymbol{r_2} \hspace{0.5em} \boldsymbol{t} \right ] \\ \boldsymbol{K}^{-1} \boldsymbol{H} &amp;= \lambda \left[ \boldsymbol{r_1} \hspace{0.5em} \boldsymbol{r_2} \hspace{0.5em} \boldsymbol{t} \right ] \\ \boldsymbol{P} &amp;= \boldsymbol{K} \left[ \boldsymbol{r_1} \hspace{0.5em} \boldsymbol{r_2} \hspace{0.5em} \left( \boldsymbol{r_1} \times \boldsymbol{r_2} \right ) \hspace{0.5em} \boldsymbol{t} \right ] \end{align*} \]
</p>
<p>This is a quick solution (see also <a class="el" href="../../d9/dab/tutorial_homography.html#projective_transformations">2</a>) as this does not ensure that the resulting rotation matrix will be orthogonal and the scale is estimated roughly by normalize the first column to 1.</p>
<p>A solution to have a proper rotation matrix (with the properties of a rotation matrix) consists to apply a polar decomposition (see <a class="el" href="../../d9/dab/tutorial_homography.html#polar_decomposition">6</a> or <a class="el" href="../../d9/dab/tutorial_homography.html#polar_decomposition_svd">7</a> for some information):</p>
<div class="fragment"><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;R (before polar decomposition):\n&quot;</span> &lt;&lt; R &lt;&lt; <span class="stringliteral">&quot;\ndet(R): &quot;</span> &lt;&lt; <a class="code" href="../../dc/d84/group__core__basic.html#ga06b8ec936c3cbc9502d76c7818053b41">determinant</a>(R) &lt;&lt; endl;</div><div class="line">    Mat W, U, Vt;</div><div class="line">    <a class="code" href="../../d2/de8/group__core__array.html#gab477b5b7b39b370bb03e75b19d2d5109">SVDecomp</a>(R, W, U, Vt);</div><div class="line">    R = U*Vt;</div><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;R (after polar decomposition):\n&quot;</span> &lt;&lt; R &lt;&lt; <span class="stringliteral">&quot;\ndet(R): &quot;</span> &lt;&lt; <a class="code" href="../../dc/d84/group__core__basic.html#ga06b8ec936c3cbc9502d76c7818053b41">determinant</a>(R) &lt;&lt; endl;</div></div><!-- fragment --><p> To check the result, the object frame projected into the image with the estimated camera pose is displayed:</p>
<div class="image">
<img src="../../homography_pose.jpg" alt="homography_pose.jpg"/>
</div>
<h3><a class="anchor" id="tutorial_homography_Demo2"></a>
Demo 2: Perspective correction</h3>
<p>In this example, a source image will be transformed into a desired perspective view by computing the homography that maps the source points into the desired points. The following image shows the source image (left) and the chessboard view that we want to transform into the desired chessboard view (right).</p>
<div class="image">
<img src="../../homography_source_desired_images.jpg" alt="homography_source_desired_images.jpg"/>
<div class="caption">
Source and desired views</div></div>
<p> The first step consists to detect the chessboard corners in the source and desired images:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    vector&lt;Point2f&gt; corners1, corners2;</div><div class="line">    <span class="keywordtype">bool</span> found1 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">findChessboardCorners</a>(img1, patternSize, corners1);</div><div class="line">    <span class="keywordtype">bool</span> found2 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">findChessboardCorners</a>(img2, patternSize, corners2);</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    ret1, corners1 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">cv.findChessboardCorners</a>(img1, patternSize)</div><div class="line">    ret2, corners2 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">cv.findChessboardCorners</a>(img2, patternSize)</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        MatOfPoint2f corners1 = <span class="keyword">new</span> MatOfPoint2f(), corners2 = <span class="keyword">new</span> MatOfPoint2f();</div><div class="line">        <span class="keywordtype">boolean</span> found1 = Calib3d.findChessboardCorners(img1, <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(9, 6), corners1 );</div><div class="line">        <span class="keywordtype">boolean</span> found2 = Calib3d.findChessboardCorners(img2, <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(9, 6), corners2 );</div></div><!-- fragment --> </div> <p>The homography is estimated easily with:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat H = <a class="code" href="../../d9/d0c/group__calib3d.html#ga4abc2ece9fab9398f2e560d53c8c9780">findHomography</a>(corners1, corners2);</div><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;H:\n&quot;</span> &lt;&lt; H &lt;&lt; endl;</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    H, _ = <a class="code" href="../../d9/d0c/group__calib3d.html#ga4b3841447530523e5272ec05c5d1e411">cv.findHomography</a>(corners1, corners2)</div><div class="line">    <a class="code" href="../../df/d57/namespacecv_1_1dnn.html#a701210a0203f2786cbfd04b2bd56da47">print</a>(H)</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Mat H = <span class="keyword">new</span> Mat();</div><div class="line">        H = Calib3d.findHomography(corners1, corners2);</div><div class="line">        System.out.println(H.dump());</div></div><!-- fragment --> </div> <p>To warp the source chessboard view into the desired chessboard view, we use <a class="el" href="../../da/d54/group__imgproc__transform.html#gaf73673a7e8e18ec6963e3774e6a94b87">cv::warpPerspective</a></p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat img1_warp;</div><div class="line">    <a class="code" href="../../d5/df1/group__imgproc__hal__functions.html#ga843a5226c302e59fe09b08f424e08795">warpPerspective</a>(img1, img1_warp, H, img1.size());</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    img1_warp = <a class="code" href="../../da/d54/group__imgproc__transform.html#gaf73673a7e8e18ec6963e3774e6a94b87">cv.warpPerspective</a>(img1, H, (img1.shape[1], img1.shape[0]))</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Mat img1_warp = <span class="keyword">new</span> Mat();</div><div class="line">        Imgproc.warpPerspective(img1, img1_warp, H, img1.size());</div></div><!-- fragment --> </div> <p>The result image is:</p>
<div class="image">
<img src="../../homography_perspective_correction_chessboard_warp.jpg" alt="homography_perspective_correction_chessboard_warp.jpg"/>
</div>
<p>To compute the coordinates of the source corners transformed by the homography:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat img_draw_matches;</div><div class="line">    <a class="code" href="../../d2/de8/group__core__array.html#gaf9771c991763233866bf76b5b5d1776f">hconcat</a>(img1, img2, img_draw_matches);</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 0; i &lt; corners1.size(); i++)</div><div class="line">    {</div><div class="line">        Mat pt1 = (Mat_&lt;double&gt;(3,1) &lt;&lt; corners1[i].x, corners1[i].y, 1);</div><div class="line">        Mat pt2 = H * pt1;</div><div class="line">        pt2 /= pt2.at&lt;<span class="keywordtype">double</span>&gt;(2);</div><div class="line"></div><div class="line">        <a class="code" href="../../dc/d84/group__core__basic.html#ga1e83eafb2d26b3c93f09e8338bcab192">Point</a> end( (<span class="keywordtype">int</span>) (img1.cols + pt2.at&lt;<span class="keywordtype">double</span>&gt;(0)), (<span class="keywordtype">int</span>) pt2.at&lt;<span class="keywordtype">double</span>&gt;(1) );</div><div class="line">        <a class="code" href="../../d6/d6e/group__imgproc__draw.html#ga7078a9fae8c7e7d13d24dac2520ae4a2">line</a>(img_draw_matches, corners1[i], end, randomColor(rng), 2);</div><div class="line">    }</div><div class="line"></div><div class="line">    <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;Draw matches&quot;</span>, img_draw_matches);</div><div class="line">    <a class="code" href="../../d7/dfc/group__highgui.html#ga5628525ad33f52eab17feebcfba38bd7">waitKey</a>();</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    img_draw_matches = <a class="code" href="../../d2/de8/group__core__array.html#ga4676b1376cdc4e528dab6bd9edc51c1a">cv.hconcat</a>([img1, img2])</div><div class="line">    <span class="keywordflow">for</span> i <span class="keywordflow">in</span> range(len(corners1)):</div><div class="line">        pt1 = np.array([corners1[i][0], corners1[i][1], 1])</div><div class="line">        pt1 = pt1.reshape(3, 1)</div><div class="line">        pt2 = np.dot(H, pt1)</div><div class="line">        pt2 = pt2/pt2[2]</div><div class="line">        end = (int(img1.shape[1] + pt2[0]), int(pt2[1]))</div><div class="line">        <a class="code" href="../../d6/d6e/group__imgproc__draw.html#ga7078a9fae8c7e7d13d24dac2520ae4a2">cv.line</a>(img_draw_matches, tuple([int(j) <span class="keywordflow">for</span> j <span class="keywordflow">in</span> corners1[i]]), end, randomColor(), 2)</div><div class="line"></div><div class="line">    <a class="code" href="../../df/d24/group__highgui__opengl.html#gaae7e90aa3415c68dba22a5ff2cefc25d">cv.imshow</a>(<span class="stringliteral">&quot;Draw matches&quot;</span>, img_draw_matches)</div><div class="line">    <a class="code" href="../../d7/dfc/group__highgui.html#ga5628525ad33f52eab17feebcfba38bd7">cv.waitKey</a>(0)</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Mat img_draw_matches = <span class="keyword">new</span> Mat();</div><div class="line">        list2.add(img1);</div><div class="line">        list2.add(img2);</div><div class="line">        Core.hconcat(list2, img_draw_matches);</div><div class="line">        <a class="code" href="../../dc/d84/group__core__basic.html#ga1e83eafb2d26b3c93f09e8338bcab192">Point</a> []corners1Arr = corners1.toArray();</div><div class="line"></div><div class="line">        <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0 ; i &lt; corners1Arr.length; i++) {</div><div class="line">            Mat pt1 = <span class="keyword">new</span> Mat(3, 1, CvType.CV_64FC1), pt2 = <span class="keyword">new</span> Mat();</div><div class="line">            pt1.put(0, 0, corners1Arr[i].x, corners1Arr[i].y, 1 );</div><div class="line"></div><div class="line">            Core.gemm(H, pt1, 1, <span class="keyword">new</span> Mat(), 0, pt2);</div><div class="line">            <span class="keywordtype">double</span>[] data = pt2.get(2, 0);</div><div class="line">            Core.divide(pt2, <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga599fe92e910c027be274233eccad7beb">Scalar</a>(data[0]), pt2);</div><div class="line"></div><div class="line">            <span class="keywordtype">double</span>[] data1 =pt2.get(0, 0);</div><div class="line">            <span class="keywordtype">double</span>[] data2 = pt2.get(1, 0);</div><div class="line">            <a class="code" href="../../dc/d84/group__core__basic.html#ga1e83eafb2d26b3c93f09e8338bcab192">Point</a> end = <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga1e83eafb2d26b3c93f09e8338bcab192">Point</a>((<span class="keywordtype">int</span>)(img1.cols()+ data1[0]), (<span class="keywordtype">int</span>)data2[0]);</div><div class="line">            Imgproc.line(img_draw_matches, corners1Arr[i], end, RandomColor(), 2);</div><div class="line">        }</div><div class="line"></div><div class="line">        HighGui.imshow(<span class="stringliteral">&quot;Draw matches&quot;</span>, img_draw_matches);</div><div class="line">        HighGui.waitKey(0);</div></div><!-- fragment --> </div> <p>To check the correctness of the calculation, the matching lines are displayed:</p>
<div class="image">
<img src="../../homography_perspective_correction_chessboard_matches.jpg" alt="homography_perspective_correction_chessboard_matches.jpg"/>
</div>
<h3><a class="anchor" id="tutorial_homography_Demo3"></a>
Demo 3: Homography from the camera displacement</h3>
<p>The homography relates the transformation between two planes and it is possible to retrieve the corresponding camera displacement that allows to go from the first to the second plane view (see <a class="el" href="../../d0/de3/citelist.html#CITEREF_Malis">[161]</a> for more information). Before going into the details that allow to compute the homography from the camera displacement, some recalls about camera pose and homogeneous transformation.</p>
<p>The function <a class="el" href="../../d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d">cv::solvePnP</a> allows to compute the camera pose from the correspondences 3D object points (points expressed in the object frame) and the projected 2D image points (object points viewed in the image). The intrinsic parameters and the distortion coefficients are required (see the camera calibration process).</p>
<p class="formulaDsp">
\[ \begin{align*} s \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} &amp;= \begin{bmatrix} f_x &amp; 0 &amp; c_x \\ 0 &amp; f_y &amp; c_y \\ 0 &amp; 0 &amp; 1 \end{bmatrix} \begin{bmatrix} r_{11} &amp; r_{12} &amp; r_{13} &amp; t_x \\ r_{21} &amp; r_{22} &amp; r_{23} &amp; t_y \\ r_{31} &amp; r_{32} &amp; r_{33} &amp; t_z \end{bmatrix} \begin{bmatrix} X_o \\ Y_o \\ Z_o \\ 1 \end{bmatrix} \\ &amp;= \boldsymbol{K} \hspace{0.2em} ^{c}\textrm{M}_o \begin{bmatrix} X_o \\ Y_o \\ Z_o \\ 1 \end{bmatrix} \end{align*} \]
</p>
<p>\( \boldsymbol{K} \) is the intrinsic matrix and \( ^{c}\textrm{M}_o \) is the camera pose. The output of <a class="el" href="../../d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d">cv::solvePnP</a> is exactly this: <code>rvec</code> is the Rodrigues rotation vector and <code>tvec</code> the translation vector.</p>
<p>\( ^{c}\textrm{M}_o \) can be represented in a homogeneous form and allows to transform a point expressed in the object frame into the camera frame:</p>
<p class="formulaDsp">
\[ \begin{align*} \begin{bmatrix} X_c \\ Y_c \\ Z_c \\ 1 \end{bmatrix} &amp;= \hspace{0.2em} ^{c}\textrm{M}_o \begin{bmatrix} X_o \\ Y_o \\ Z_o \\ 1 \end{bmatrix} \\ &amp;= \begin{bmatrix} ^{c}\textrm{R}_o &amp; ^{c}\textrm{t}_o \\ 0_{1\times3} &amp; 1 \end{bmatrix} \begin{bmatrix} X_o \\ Y_o \\ Z_o \\ 1 \end{bmatrix} \\ &amp;= \begin{bmatrix} r_{11} &amp; r_{12} &amp; r_{13} &amp; t_x \\ r_{21} &amp; r_{22} &amp; r_{23} &amp; t_y \\ r_{31} &amp; r_{32} &amp; r_{33} &amp; t_z \\ 0 &amp; 0 &amp; 0 &amp; 1 \end{bmatrix} \begin{bmatrix} X_o \\ Y_o \\ Z_o \\ 1 \end{bmatrix} \end{align*} \]
</p>
<p>Transform a point expressed in one frame to another frame can be easily done with matrix multiplication:</p>
<ul>
<li>\( ^{c_1}\textrm{M}_o \) is the camera pose for the camera 1</li>
<li>\( ^{c_2}\textrm{M}_o \) is the camera pose for the camera 2</li>
</ul>
<p>To transform a 3D point expressed in the camera 1 frame to the camera 2 frame:</p>
<p class="formulaDsp">
\[ ^{c_2}\textrm{M}_{c_1} = \hspace{0.2em} ^{c_2}\textrm{M}_{o} \cdot \hspace{0.1em} ^{o}\textrm{M}_{c_1} = \hspace{0.2em} ^{c_2}\textrm{M}_{o} \cdot \hspace{0.1em} \left( ^{c_1}\textrm{M}_{o} \right )^{-1} = \begin{bmatrix} ^{c_2}\textrm{R}_{o} &amp; ^{c_2}\textrm{t}_{o} \\ 0_{3 \times 1} &amp; 1 \end{bmatrix} \cdot \begin{bmatrix} ^{c_1}\textrm{R}_{o}^T &amp; - \hspace{0.2em} ^{c_1}\textrm{R}_{o}^T \cdot \hspace{0.2em} ^{c_1}\textrm{t}_{o} \\ 0_{1 \times 3} &amp; 1 \end{bmatrix} \]
</p>
<p>In this example, we will compute the camera displacement between two camera poses with respect to the chessboard object. The first step consists to compute the camera poses for the two images:</p>
<div class="fragment"><div class="line">    vector&lt;Point2f&gt; corners1, corners2;</div><div class="line">    <span class="keywordtype">bool</span> found1 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">findChessboardCorners</a>(img1, patternSize, corners1);</div><div class="line">    <span class="keywordtype">bool</span> found2 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga93efa9b0aa890de240ca32b11253dd4a">findChessboardCorners</a>(img2, patternSize, corners2);</div><div class="line"></div><div class="line">    <span class="keywordflow">if</span> (!found1 || !found2)</div><div class="line">    {</div><div class="line">        cout &lt;&lt; <span class="stringliteral">&quot;Error, cannot find the chessboard corners in both images.&quot;</span> &lt;&lt; endl;</div><div class="line">        <span class="keywordflow">return</span>;</div><div class="line">    }</div><div class="line"></div><div class="line">    vector&lt;Point3f&gt; objectPoints;</div><div class="line">    calcChessboardCorners(patternSize, squareSize, objectPoints);</div><div class="line"></div><div class="line">    FileStorage fs( <a class="code" href="../../d6/dba/group__core__utils__samples.html#ga3a33b00033b46c698ff6340d95569c13">samples::findFile</a>( intrinsicsPath ), FileStorage::READ);</div><div class="line">    Mat cameraMatrix, distCoeffs;</div><div class="line">    fs[<span class="stringliteral">&quot;camera_matrix&quot;</span>] &gt;&gt; cameraMatrix;</div><div class="line">    fs[<span class="stringliteral">&quot;distortion_coefficients&quot;</span>] &gt;&gt; distCoeffs;</div><div class="line"></div><div class="line">    Mat rvec1, tvec1;</div><div class="line">    <a class="code" href="../../d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d">solvePnP</a>(objectPoints, corners1, cameraMatrix, distCoeffs, rvec1, tvec1);</div><div class="line">    Mat rvec2, tvec2;</div><div class="line">    <a class="code" href="../../d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d">solvePnP</a>(objectPoints, corners2, cameraMatrix, distCoeffs, rvec2, tvec2);</div></div><!-- fragment --> <div class="image">
<img src="../../homography_camera_displacement_poses.jpg" alt="homography_camera_displacement_poses.jpg"/>
</div>
<p>The camera displacement can be computed from the camera poses using the formulas above:</p>
<div class="fragment"><div class="line"><span class="keywordtype">void</span> computeC2MC1(<span class="keyword">const</span> Mat &amp;R1, <span class="keyword">const</span> Mat &amp;tvec1, <span class="keyword">const</span> Mat &amp;R2, <span class="keyword">const</span> Mat &amp;tvec2,</div><div class="line">                  Mat &amp;R_1to2, Mat &amp;tvec_1to2)</div><div class="line">{</div><div class="line">    <span class="comment">//c2Mc1 = c2Mo * oMc1 = c2Mo * c1Mo.inv()</span></div><div class="line">    R_1to2 = R2 * R1.t();</div><div class="line">    tvec_1to2 = R2 * (-R1.t()*tvec1) + tvec2;</div><div class="line">}</div></div><!-- fragment --><p> The homography related to a specific plane computed from the camera displacement is:</p>
<div class="image">
<img src="../../homography_camera_displacement.png" alt="homography_camera_displacement.png"/>
<div class="caption">
By Homography-transl.svg: Per Rosengren derivative work: Appoose (Homography-transl.svg) [CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons</div></div>
<p> On this figure, <code>n</code> is the normal vector of the plane and <code>d</code> the distance between the camera frame and the plane along the plane normal. The <a href="https://en.wikipedia.org/wiki/Homography_(computer_vision)#3D_plane_to_plane_equation">equation</a> to compute the homography from the camera displacement is:</p>
<p class="formulaDsp">
\[ ^{2}\textrm{H}_{1} = \hspace{0.2em} ^{2}\textrm{R}_{1} - \hspace{0.1em} \frac{^{2}\textrm{t}_{1} \cdot n^T}{d} \]
</p>
<p>Where \( ^{2}\textrm{H}_{1} \) is the homography matrix that maps the points in the first camera frame to the corresponding points in the second camera frame, \( ^{2}\textrm{R}_{1} = \hspace{0.2em} ^{c_2}\textrm{R}_{o} \cdot \hspace{0.1em} ^{c_1}\textrm{R}_{o}^{T} \) is the rotation matrix that represents the rotation between the two camera frames and \( ^{2}\textrm{t}_{1} = \hspace{0.2em} ^{c_2}\textrm{R}_{o} \cdot \left( - \hspace{0.1em} ^{c_1}\textrm{R}_{o}^{T} \cdot \hspace{0.1em} ^{c_1}\textrm{t}_{o} \right ) + \hspace{0.1em} ^{c_2}\textrm{t}_{o} \) the translation vector between the two camera frames.</p>
<p>Here the normal vector <code>n</code> is the plane normal expressed in the camera frame 1 and can be computed as the cross product of 2 vectors (using 3 non collinear points that lie on the plane) or in our case directly with:</p>
<div class="fragment"><div class="line">    Mat normal = (Mat_&lt;double&gt;(3,1) &lt;&lt; 0, 0, 1);</div><div class="line">    Mat normal1 = R1*normal;</div></div><!-- fragment --><p> The distance <code>d</code> can be computed as the dot product between the plane normal and a point on the plane or by computing the <a href="http://mathworld.wolfram.com/Plane.html">plane equation</a> and using the D coefficient:</p>
<div class="fragment"><div class="line">    Mat origin(3, 1, <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga30a562691cc5987bc88eb7bb7a8faf2b">CV_64F</a>, <a class="code" href="../../dc/d84/group__core__basic.html#ga599fe92e910c027be274233eccad7beb">Scalar</a>(0));</div><div class="line">    Mat origin1 = R1*origin + tvec1;</div><div class="line">    <span class="keywordtype">double</span> d_inv1 = 1.0 / normal1.dot(origin1);</div></div><!-- fragment --><p> The projective homography matrix \( \textbf{G} \) can be computed from the Euclidean homography \( \textbf{H} \) using the intrinsic matrix \( \textbf{K} \) (see <a class="el" href="../../d0/de3/citelist.html#CITEREF_Malis">[161]</a>), here assuming the same camera between the two plane views:</p>
<p class="formulaDsp">
\[ \textbf{G} = \gamma \textbf{K} \textbf{H} \textbf{K}^{-1} \]
</p>
<div class="fragment"><div class="line">Mat computeHomography(<span class="keyword">const</span> Mat &amp;R_1to2, <span class="keyword">const</span> Mat &amp;tvec_1to2, <span class="keyword">const</span> <span class="keywordtype">double</span> d_inv, <span class="keyword">const</span> Mat &amp;normal)</div><div class="line">{</div><div class="line">    Mat homography = R_1to2 + d_inv * tvec_1to2*normal.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#aaa428c60ccb6d8ea5de18f63dfac8e11">t</a>();</div><div class="line">    <span class="keywordflow">return</span> homography;</div><div class="line">}</div></div><!-- fragment --><p> In our case, the Z-axis of the chessboard goes inside the object whereas in the homography figure it goes outside. This is just a matter of sign:</p>
<p class="formulaDsp">
\[ ^{2}\textrm{H}_{1} = \hspace{0.2em} ^{2}\textrm{R}_{1} + \hspace{0.1em} \frac{^{2}\textrm{t}_{1} \cdot n^T}{d} \]
</p>
<div class="fragment"><div class="line">    Mat homography_euclidean = computeHomography(R_1to2, t_1to2, d_inv1, normal1);</div><div class="line">    Mat homography = cameraMatrix * homography_euclidean * cameraMatrix.inv();</div><div class="line"></div><div class="line">    homography /= homography.at&lt;<span class="keywordtype">double</span>&gt;(2,2);</div><div class="line">    homography_euclidean /= homography_euclidean.at&lt;<span class="keywordtype">double</span>&gt;(2,2);</div></div><!-- fragment --><p> We will now compare the projective homography computed from the camera displacement with the one estimated with <a class="el" href="../../d9/d0c/group__calib3d.html#ga4abc2ece9fab9398f2e560d53c8c9780">cv::findHomography</a></p>
<div class="fragment"><div class="line">findHomography H:</div><div class="line">[0.32903393332201, -1.244138808862929, 536.4769088231476;</div><div class="line"> 0.6969763913334046, -0.08935909072571542, -80.34068504082403;</div><div class="line"> 0.00040511729592961, -0.001079740100565013, 0.9999999999999999]</div><div class="line"></div><div class="line">homography from camera displacement:</div><div class="line">[0.4160569997384721, -1.306889006892538, 553.7055461075881;</div><div class="line"> 0.7917584252773352, -0.06341244158456338, -108.2770029401219;</div><div class="line"> 0.0005926357240956578, -0.001020651672127799, 1]</div></div><!-- fragment --><p>The homography matrices are similar. If we compare the image 1 warped using both homography matrices:</p>
<div class="image">
<img src="../../homography_camera_displacement_compare.jpg" alt="homography_camera_displacement_compare.jpg"/>
<div class="caption">
Left: image warped using the homography estimated. Right: using the homography computed from the camera displacement</div></div>
<p> Visually, it is hard to distinguish a difference between the result image from the homography computed from the camera displacement and the one estimated with <a class="el" href="../../d9/d0c/group__calib3d.html#ga4abc2ece9fab9398f2e560d53c8c9780">cv::findHomography</a> function.</p>
<h3><a class="anchor" id="tutorial_homography_Demo4"></a>
Demo 4: Decompose the homography matrix</h3>
<p>OpenCV 3 contains the function <a class="el" href="../../d9/d0c/group__calib3d.html#ga7f60bdff78833d1e3fd6d9d0fd538d92">cv::decomposeHomographyMat</a> which allows to decompose the homography matrix to a set of rotations, translations and plane normals. First we will decompose the homography matrix computed from the camera displacement:</p>
<div class="fragment"><div class="line">    Mat homography_euclidean = computeHomography(R_1to2, t_1to2, d_inv1, normal1);</div><div class="line">    Mat homography = cameraMatrix * homography_euclidean * cameraMatrix.inv();</div><div class="line"></div><div class="line">    homography /= homography.at&lt;<span class="keywordtype">double</span>&gt;(2,2);</div><div class="line">    homography_euclidean /= homography_euclidean.at&lt;<span class="keywordtype">double</span>&gt;(2,2);</div></div><!-- fragment --><p> The results of <a class="el" href="../../d9/d0c/group__calib3d.html#ga7f60bdff78833d1e3fd6d9d0fd538d92">cv::decomposeHomographyMat</a> are:</p>
<div class="fragment"><div class="line">    vector&lt;Mat&gt; Rs_decomp, ts_decomp, normals_decomp;</div><div class="line">    <span class="keywordtype">int</span> solutions = <a class="code" href="../../d9/d0c/group__calib3d.html#ga7f60bdff78833d1e3fd6d9d0fd538d92">decomposeHomographyMat</a>(homography, cameraMatrix, Rs_decomp, ts_decomp, normals_decomp);</div><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;Decompose homography matrix computed from the camera displacement:&quot;</span> &lt;&lt; endl &lt;&lt; endl;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0; i &lt; solutions; i++)</div><div class="line">    {</div><div class="line">      <span class="keywordtype">double</span> factor_d1 = 1.0 / d_inv1;</div><div class="line">      Mat rvec_decomp;</div><div class="line">      <a class="code" href="../../d9/d0c/group__calib3d.html#ga61585db663d9da06b68e70cfbf6a1eac">Rodrigues</a>(Rs_decomp[i], rvec_decomp);</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;Solution &quot;</span> &lt;&lt; i &lt;&lt; <span class="stringliteral">&quot;:&quot;</span> &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;rvec from homography decomposition: &quot;</span> &lt;&lt; rvec_decomp.t() &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;rvec from camera displacement: &quot;</span> &lt;&lt; rvec_1to2.t() &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;tvec from homography decomposition: &quot;</span> &lt;&lt; ts_decomp[i].t() &lt;&lt; <span class="stringliteral">&quot; and scaled by d: &quot;</span> &lt;&lt; factor_d1 * ts_decomp[i].t() &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;tvec from camera displacement: &quot;</span> &lt;&lt; t_1to2.t() &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;plane normal from homography decomposition: &quot;</span> &lt;&lt; normals_decomp[i].t() &lt;&lt; endl;</div><div class="line">      cout &lt;&lt; <span class="stringliteral">&quot;plane normal at camera 1 pose: &quot;</span> &lt;&lt; normal1.t() &lt;&lt; endl &lt;&lt; endl;</div><div class="line">    }</div></div><!-- fragment --> <div class="fragment"><div class="line">Solution 0:</div><div class="line">rvec from homography decomposition: [-0.0919829920641369, -0.5372581036567992, 1.310868863540717]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [-0.7747961019053186, -0.02751124463434032, -0.6791980037590677] and scaled by d: [-0.1578091561210742, -0.005603443652993778, -0.1383378976078466]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [-0.1973513139420648, 0.6283451996579074, -0.7524857267431757]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 1:</div><div class="line">rvec from homography decomposition: [-0.0919829920641369, -0.5372581036567992, 1.310868863540717]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [0.7747961019053186, 0.02751124463434032, 0.6791980037590677] and scaled by d: [0.1578091561210742, 0.005603443652993778, 0.1383378976078466]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [0.1973513139420648, -0.6283451996579074, 0.7524857267431757]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 2:</div><div class="line">rvec from homography decomposition: [0.1053487907109967, -0.1561929144786397, 1.401356552358475]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [-0.4666552552894618, 0.1050032934770042, -0.913007654671646] and scaled by d: [-0.0950475510338766, 0.02138689274867372, -0.1859598508065552]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [-0.3131715472900788, 0.8421206145721947, -0.4390403768225507]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 3:</div><div class="line">rvec from homography decomposition: [0.1053487907109967, -0.1561929144786397, 1.401356552358475]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [0.4666552552894618, -0.1050032934770042, 0.913007654671646] and scaled by d: [0.0950475510338766, -0.02138689274867372, 0.1859598508065552]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [0.3131715472900788, -0.8421206145721947, 0.4390403768225507]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div></div><!-- fragment --><p>The result of the decomposition of the homography matrix can only be recovered up to a scale factor that corresponds in fact to the distance <code>d</code> as the normal is unit length. As you can see, there is one solution that matches almost perfectly with the computed camera displacement. As stated in the documentation:</p>
<div class="fragment"><div class="line">At least two of the solutions may further be invalidated if point correspondences are available by applying positive depth constraint (all points must be in front of the camera).</div></div><!-- fragment --><p>As the result of the decomposition is a camera displacement, if we have the initial camera pose \( ^{c_1}\textrm{M}_{o} \), we can compute the current camera pose \( ^{c_2}\textrm{M}_{o} = \hspace{0.2em} ^{c_2}\textrm{M}_{c_1} \cdot \hspace{0.1em} ^{c_1}\textrm{M}_{o} \) and test if the 3D object points that belong to the plane are projected in front of the camera or not. Another solution could be to retain the solution with the closest normal if we know the plane normal expressed at the camera 1 pose.</p>
<p>The same thing but with the homography matrix estimated with <a class="el" href="../../d9/d0c/group__calib3d.html#ga4abc2ece9fab9398f2e560d53c8c9780">cv::findHomography</a></p>
<div class="fragment"><div class="line">Solution 0:</div><div class="line">rvec from homography decomposition: [0.1552207729599141, -0.152132696119647, 1.323678695078694]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [-0.4482361704818117, 0.02485247635491922, -1.034409687207331] and scaled by d: [-0.09129598307571339, 0.005061910238634657, -0.2106868109173855]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [-0.1384902722707529, 0.9063331452766947, -0.3992250922214516]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 1:</div><div class="line">rvec from homography decomposition: [0.1552207729599141, -0.152132696119647, 1.323678695078694]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [0.4482361704818117, -0.02485247635491922, 1.034409687207331] and scaled by d: [0.09129598307571339, -0.005061910238634657, 0.2106868109173855]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [0.1384902722707529, -0.9063331452766947, 0.3992250922214516]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 2:</div><div class="line">rvec from homography decomposition: [-0.2886605671759886, -0.521049903923871, 1.381242030882511]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [-0.8705961357284295, 0.1353018038908477, -0.7037702049789747] and scaled by d: [-0.177321544550518, 0.02755804196893467, -0.1433427218822783]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [-0.2284582117722427, 0.6009247303964522, -0.7659610393954643]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div><div class="line"></div><div class="line">Solution 3:</div><div class="line">rvec from homography decomposition: [-0.2886605671759886, -0.521049903923871, 1.381242030882511]</div><div class="line">rvec from camera displacement: [-0.09198299206413783, -0.5372581036567995, 1.310868863540717]</div><div class="line">tvec from homography decomposition: [0.8705961357284295, -0.1353018038908477, 0.7037702049789747] and scaled by d: [0.177321544550518, -0.02755804196893467, 0.1433427218822783]</div><div class="line">tvec from camera displacement: [0.1578091561210745, 0.005603443652993617, 0.1383378976078466]</div><div class="line">plane normal from homography decomposition: [0.2284582117722427, -0.6009247303964522, 0.7659610393954643]</div><div class="line">plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.752485726743176]</div></div><!-- fragment --><p>Again, there is also a solution that matches with the computed camera displacement.</p>
<h3><a class="anchor" id="tutorial_homography_Demo5"></a>
Demo 5: Basic panorama stitching from a rotating camera</h3>
<dl class="section note"><dt>Note</dt><dd>This example is made to illustrate the concept of image stitching based on a pure rotational motion of the camera and should not be used to stitch panorama images. The <a class="el" href="../../d1/d46/group__stitching.html">stitching module</a> provides a complete pipeline to stitch images.</dd></dl>
<p>The homography transformation applies only for planar structure. But in the case of a rotating camera (pure rotation around the camera axis of projection, no translation), an arbitrary world can be considered (<a class="el" href="../../d9/dab/tutorial_homography.html#tutorial_homography_What_is_the_homography_matrix">see previously</a>).</p>
<p>The homography can then be computed using the rotation transformation and the camera intrinsic parameters as (see for instance <a class="el" href="../../d9/dab/tutorial_homography.html#homography_course">8</a>):</p>
<p class="formulaDsp">
\[ s \begin{bmatrix} x^{&#39;} \\ y^{&#39;} \\ 1 \end{bmatrix} = \bf{K} \hspace{0.1em} \bf{R} \hspace{0.1em} \bf{K}^{-1} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \]
</p>
<p>To illustrate, we used Blender, a free and open-source 3D computer graphics software, to generate two camera views with only a rotation transformation between each other. More information about how to retrieve the camera intrinsic parameters and the <code>3x4</code> extrinsic matrix with respect to the world can be found in <a class="el" href="../../d9/dab/tutorial_homography.html#answer_blender">9</a> (an additional transformation is needed to get the transformation between the camera and the object frames) with Blender.</p>
<p>The figure below shows the two generated views of the Suzanne model, with only a rotation transformation:</p>
<div class="image">
<img src="../../homography_stitch_compare.jpg" alt="homography_stitch_compare.jpg"/>
</div>
<p>With the known associated camera poses and the intrinsic parameters, the relative rotation between the two views can be computed:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat R1 = c1Mo(Range(0,3), Range(0,3));</div><div class="line">    Mat R2 = c2Mo(Range(0,3), Range(0,3));</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    R1 = c1Mo[0:3, 0:3]</div><div class="line">    R2 = c2Mo[0:3, 0:3]</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Range rowRange = <span class="keyword">new</span> Range(0,3);</div><div class="line">        Range colRange = <span class="keyword">new</span> Range(0,3);</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    <span class="comment">//c1Mo * oMc2</span></div><div class="line">    Mat R_2to1 = R1*R2.t();</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    R2 = R2.transpose()</div><div class="line">    R_2to1 = np.dot(R1,R2)</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        <span class="comment">//c1Mo * oMc2</span></div><div class="line">        Mat R1 = <span class="keyword">new</span>  Mat(c1Mo, rowRange, colRange);</div><div class="line">        Mat R2 = <span class="keyword">new</span> Mat(c2Mo, rowRange, colRange);</div><div class="line">        Mat R_2to1 = <span class="keyword">new</span> Mat();</div><div class="line">        Core.gemm(R1, R2.t(), 1, <span class="keyword">new</span> Mat(), 0, R_2to1 );</div></div><!-- fragment --> </div> <p>Here, the second image will be stitched with respect to the first image. The homography can be calculated using the formula above:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat H = cameraMatrix * R_2to1 * cameraMatrix.inv();</div><div class="line">    H /= H.at&lt;<span class="keywordtype">double</span>&gt;(2,2);</div><div class="line">    cout &lt;&lt; <span class="stringliteral">&quot;H:\n&quot;</span> &lt;&lt; H &lt;&lt; endl;</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    H = cameraMatrix.dot(R_2to1).dot(np.linalg.inv(cameraMatrix))</div><div class="line">    H = H / H[2][2]</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Mat tmp = <span class="keyword">new</span> Mat(), H = <span class="keyword">new</span> Mat();</div><div class="line">        Core.gemm(cameraMatrix, R_2to1, 1, <span class="keyword">new</span> Mat(), 0, tmp);</div><div class="line">        Core.gemm(tmp, cameraMatrix.inv(), 1, <span class="keyword">new</span> Mat(), 0, H);</div><div class="line">        <a class="code" href="../../dc/d84/group__core__basic.html#ga599fe92e910c027be274233eccad7beb">Scalar</a> s = <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga599fe92e910c027be274233eccad7beb">Scalar</a>(H.get(2, 2)[0]);</div><div class="line">        Core.divide(H, s, H);</div><div class="line">        System.out.println(H.dump());</div></div><!-- fragment --> </div> <p>The stitching is made simply with:</p>
 <div class='newInnerHTML' title='cpp' style='display: none;'>C++</div><div class='toggleable_div label_cpp' style='display: none;'> <div class="fragment"><div class="line">    Mat img_stitch;</div><div class="line">    <a class="code" href="../../d5/df1/group__imgproc__hal__functions.html#ga843a5226c302e59fe09b08f424e08795">warpPerspective</a>(img2, img_stitch, H, <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(img2.cols*2, img2.rows));</div><div class="line">    Mat half = img_stitch(<a class="code" href="../../dc/d84/group__core__basic.html#ga11d95de507098e90bad732b9345402e8">Rect</a>(0, 0, img1.cols, img1.rows));</div><div class="line">    img1.copyTo(half);</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='python' style='display: none;'>Python</div><div class='toggleable_div label_python' style='display: none;'> <div class="fragment"><div class="line">    img_stitch = <a class="code" href="../../da/d54/group__imgproc__transform.html#gaf73673a7e8e18ec6963e3774e6a94b87">cv.warpPerspective</a>(img2, H, (img2.shape[1]*2, img2.shape[0]))</div><div class="line">    img_stitch[0:img1.shape[0], 0:img1.shape[1]] = img1</div></div><!-- fragment --> </div>  <div class='newInnerHTML' title='java' style='display: none;'>Java</div><div class='toggleable_div label_java' style='display: none;'> <div class="fragment"><div class="line">        Mat img_stitch = <span class="keyword">new</span> Mat();</div><div class="line">        Imgproc.warpPerspective(img2, img_stitch, H, <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(img2.cols()*2, img2.rows()) );</div><div class="line">        Mat half = <span class="keyword">new</span> Mat();</div><div class="line">        half =  <span class="keyword">new</span> Mat(img_stitch, <span class="keyword">new</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga11d95de507098e90bad732b9345402e8">Rect</a>(0, 0, img1.cols(), img1.rows()));</div><div class="line">        img1.copyTo(half);</div></div><!-- fragment --> </div> <p>The resulting image is:</p>
<div class="image">
<img src="../../homography_stitch_Suzanne.jpg" alt="homography_stitch_Suzanne.jpg"/>
</div>
<h2><a class="anchor" id="tutorial_homography_Additional_references"></a>
Additional references </h2>
<ul>
<li><a class="anchor" id="lecture_16"></a>1. <a href="http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf">Lecture 16: Planar Homographies</a>, Robert Collins</li>
<li><a class="anchor" id="projective_transformations"></a>2. <a href="https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws11-12/3DCV_WS11-12_lec04.pdf">2D projective transformations (homographies)</a>, Christiano Gava, Gabriele Bleser</li>
<li><a class="anchor" id="szeliski"></a>3. <a href="http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf">Computer Vision: Algorithms and Applications</a>, Richard Szeliski</li>
<li><a class="anchor" id="answer_dsp"></a>4. <a href="https://dsp.stackexchange.com/a/2737">Step by Step Camera Pose Estimation for Visual Tracking and Planar Markers</a></li>
<li><a class="anchor" id="pose_ar"></a>5. <a href="https://team.inria.fr/lagadic/camera_localization/tutorial-pose-dlt-planar-opencv.html">Pose from homography estimation</a></li>
<li><a class="anchor" id="polar_decomposition"></a>6. <a href="http://www.continuummechanics.org/polardecomposition.html">Polar Decomposition (in Continuum Mechanics)</a></li>
<li><a class="anchor" id="polar_decomposition_svd"></a>7. <a href="https://web.stanford.edu/~gavish/documents/SVD_ans_you.pdf">A Personal Interview with the Singular Value Decomposition</a>, Matan Gavish</li>
<li><a class="anchor" id="homography_course"></a>8. <a href="http://people.scs.carleton.ca/~c_shu/Courses/comp4900d/notes/homography.pdf">Homography</a>, Dr. Gerhard Roth</li>
<li><a class="anchor" id="answer_blender"></a>9. <a href="https://blender.stackexchange.com/a/38210">3x4 camera matrix from blender camera</a> </li>
</ul>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:34 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
